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general  properties  common  to  circuits  of  a  given  type;  and  (c)  procedural 
knowledge,  which  pertains  to  the  Way  that  a  circuit  can  be  modified  and  to  the 
interaction  among  knowledge  elements  of  all  three  types  of  knowledge.  Early 
phases  df  this  work  focused  on  a  study  of  individual  differences  in  structural 
knowledge  and  an  experiment  conducted  to  investigate  individual  differences  in 
procedural  knowledge.  Novice  and  expert  subjects  performed  tasks  in  which  they 
had  to  either  locate  and  correct  an  error  in  a  circuit,  change  the  function 
of  a  circuit,  or  complete  a  missing  segment  in  a  circuit.  On  all  tasks,  experts 
were  found  to  be  far  more  accurate  than  novices;  but  more  important,  experts  were 
classified  --  on  the  basis  of  verbal  protocols  --  to  be  considerably  more 
systematic,  orderly,  and  directed  in  their  problem  solving  strategies.  The 
productive  procedures  used  by  experts  were  then  translated  into  specific  guideline 
toward  improving  circuit  comprehension  and  troubleshooting,  and  the  effectiveness 
of  these  guidelines  were  evaluated  in  a  subsequent  experiment.  The  results  of  thi 
research  program  should  help  in  providing  guidelines  for  training  electronic 
techniques  to  better  understand  and  troubleshoot  complex  equipment. 
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ABSTRACT 


This  work  Is  concerned  with  the  knowledge  that  electronics  technicians 
possess  of  electronic  equipment,  and  more  generally,  with  how  people 
operate  In  tasks  that  draw  upon  a  complex  spatial  symbolic  knowledge  base. 

A  technician's  knowledge  base  Is  postulated  to  consist  of  three  types  of 
related  knowledge:  (a)  structural /functional  knowledge,  which  pertains  to 
the  actual  configuration  of  a  circuit  and  the  role  that  Its  components  play 
In  the  operation  of  the  device;  (b)  prototypical  knowledge,  which  pertains 
to  the  general  properties  common  to  circuits  of  a  given  type;  and  (c) 
procedural  knowledge,  which  pertains  to  the  way  that  a  circuit  can  be 
modified  and  to  the  Interaction  among  knowledge  elements  of  all  three  types 
of  knowledge.  Early  phases  of  this  work  focused  on  a  study  of  Individual 
differences  In  structural  knowledge  and  an  experiment  conducted  to 
Investigate  Individual  differences  In  procedural  knowledge.  Novice  and 
expert  subjects  performed  tasks  In  which  they  had  to  either  locate  and 
correct  an  error  In  a  circuit,  change  the  function  of  a  circuit,  or 
complete  a  missing  segment  in  a  circuit.  On  all  tasks,  experts  were  found 
to  be  far  more  accurate  than  novices;  but  more  Important,  experts  were 
classified  —  on  the  basis  of  verbal  protocols  —  to  be  considerably  more 
systematic,  orderly,  and  directed  In  their  problem  solving  strategies.  The 
productive  procedures  used  by  experts  were  then  translated  Into  specific 
guidelines  toward  Improving  circuit  comprehension  and  troubleshooting,  and 
the  effectiveness  of  these  guidelines  were  evaluated  in  a  subsequent 
experiment.  The  results  of  this  research  program  should  help  in  providing 
guidelines  for  training  electronic  techniques  to  better  understand  and 
troubleshoot  complex  equipment. 


1.  INTRODUCTION 


1.1  Summary 

This  report  describes  a  three-year  program  of  research  devoted  to 
understanding  the  knowledge  base  required  by  experienced  technicians  to 
troubleshoot  complex  electronic  equipment.  The  primary  objectives  of  this 
program  were  to  (1)  describe  the  mental  representations  of  electronic 
devices  that  technicians  derive  from  schematic  circuit  diagrams,  and  (2) 
characterize  the  ways  In  which  technicians  apply  these  mental 
representations  to  trouble-shooting  and  problem  solving.  In  the  first 
year,  we  largely  accomplished  the  first  objective  with  a  macro-experiment 
designed  to  assess  the  structural /functional  knowledge  of  electronic 
devices  possessed  by  technicians  varying  In  skill  level.  Observed 
performance  differences  Included  errors  In  characteristic  places  In  the 
circuits  and  global  differences  In  the  structure  and  organization  of  the 
knowledge  base. 

It  was  our  view  that  these  differences,  which  primarily  reflect  knowledge 
of  facts  about  the  circuit,  were  not  sufficient  to  explain  the  variations 
In  skill  between  technicians,  nor  are  these  differences  the  most 
Interesting  for  Investigation.  In  particular,  such  differences  do  not 
address  the  way  that  the  circuit  knowledge  Is  used.  One  could  train  novice 
technicians  on  the  memorization  of  circuit  diagrams  until  they  made  few 
errors  In  retention,  but  they  would  not  become  experts  through  this 
training.  Practical  tasks  require  more  than  static  comprehension  of  a 
device.  The  knowledge  base  is  valuable  only  to  the  extent  that  its  holder 
can  operate  upon  It.  It  was  our  hypothesis  that  the  Increased  proficiency 
of  technicians  derived  through  experience  Is  due  to  the  acquisition  of  more 
sophisticated  procedural  knowledge,  not  simply  to  the  development  of  a  more 
complex  static  knowledge  base.  Accordingly,  In  the  second  year  of  this 


effort,  we  concentrated  on  the  study  of  procedural  knowledge,  with  the 
research  Information  gained  during  the  first  year  providing  the  necessary 
technical  background. 


The  third  year  of  work  focused  on  an  Integration  and  summation  of  the 
experiments  performed  during  the  first  two  years.  This  effort  Included  the 
development  and  validation  of  guidelines  for  comprehension  and 
troubleshooting  of  circuits.  Our  premise  has  been  that  experts  have  a 
mental  representation  of  their  area  of  expertise  that  Is  concordant  with 
the  Information  processing  requirements  that  operations  on  the  material 
entail.  This  representation  can  be  translated  to  some  extent  Into 
guidelines  that  can  be  used  to  Improve  the  performance  of  other  Individuals 
who  have,  as  yet,  more  limited  or  otherwise  less  coherent  mental 
representation  of  these  procedures. 

1.2  Program  Overview 

1.2.1  Objectives.  The  overall  objectives  of  the  program  Included  the 
following: 

(1)  Describe  the  mental  representation  of  electronic  devices  that 
technicians  derive  from  schematic  circuit  diagrams. 

(2)  Characterize  the  procedural  knowledge  that  technicians  apply 
to  the  mental  representation  to  perform  troubleshooting  and 
problem  solving. 

(3)  Characterize  the  knowledge  structures  and  procedures  that 
differentiate  among  technicians  of  different  skill  levels. 

(4)  Validate  the  findings  of  the  descriptive  studies  by  means  of 
hypothet 1 co-deductlve  experl mentat 1 on . 


Three-Year  Program.  The  research  tasks  that  were  conducted  during  the 
three-year  program  were  as  follows: 


(1)  First-Year:  Exploration  of  structural  and  functional 

knowledge.  Oevelop  stimulus  material  pool.  Perform  several 
studies  with  electronic  technicians  varying  In  skill  level  to 
assess  the  structural /functional  knowledge  they  possess  of 
electronic  devices  as  represented  by  circuit  diagrams. 

(2)  Second-Year:  Investigation  of  Procedural  Knowledge.  Conduct 
descriptive  studies  and  hypothesis  testing  experiments  to 
assess  the  procedural  procedural  knowledge  that  Is  brought  to 
bear  on  solving  problems  with  circuit  diagrams  by  both  novice 
and  more  proficient  electronics  technicians. 

(3)  Third-Year:  Integration  and  Summation  of  Studies  Performed 
During  First  Two  Years.  Develop  and  validate  guidelines  for 
comprehending  circuits  and  troubleshooting.  This  included  an 
experiment  suggested  by  the  findings  from  the  first  two  years. 

1.3  Overview  of  the  Technical  Approach 

An  overview  of  the  technical  approach  for  the  program  Is  shown  in  Figure 
1-1.  Work  In  the  first  year  began  with  Task  1,  the  development  of  a  pool 
of  stimulus  circuit  diagram  materials  for  use  In  all  of  the  subsequent 
work.  In  Task  2,  a  macro-experiment  was  conducted  to  reveal  the 
structural/functional  knowledge  possessed  by  electronic  technicians  varying 
In  level  of  expertise.  The  tasks  studied  In  the  first  year  were  a  memory 
task  and  a  component-partitioning  task.  In  the  second  year.  Task  3 
Investigated  the  procedural  knowledge  that  technicians  apply  to  the 
structural/functional  representation  of  electronic  circuits.  Three 
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FIGURE  1-1. 
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separate  circuit-based  tasks  were  studied  In  the  second  year:  an  error 
correction  task,  an  alteration  task,  and  a  completion  task.  Work  In  the 
third  year  focused  on  the  Interplay  between  the  structural /functional 
knowledge  and  the  procedural  knowledge,  specifically  on  the  development  and 
validation  of  guidelines  for  problem  solving  with  circuits. 


2.  RESEARCH  BACKGROUND 


2.1  Overview 


The  performance  of  complex  tasks,  such  as  maintaining  mechanical  equipment, 
modifying  and  adapting  computer  programs,  and  troubleshooting  sophisticated 
electronic  devices,  requires  highly  skilled  personnel.  All  such  tasks 
require  these  personnel  to  have  a  detailed  understanding  of  the  devices 
that  are  being  repaired  and  maintained.  Thus,  an  understanding  of  how 
people  mentally  represent  complex  devices  Is  necessary  to  train  such 
personnel  and  to  develop  optimal  man-machine  Interfaces  that  such  personnel 
can  use. 


It  Is  commonly  accepted  that  the  training  of  personnel  to  troubleshoot 
electronic  circuits  has  been  less  than  successful.  In  a  review  of  the 
status  of  troubleshooting  In  the  military  services.  Bond  and  Towne  (1979) 
state: 


The  main  conclusion  of  this  report... (Is  that)  trouble¬ 
shooting  of  very  complex  systems  Is  difficult  for 
numerous  reasons,  but  the  critical  factor  Is  that  the 
technician's  cognitive  map  of  essential  physical 
relations  (electronic,  hydraulic,  electro-mechanical,  and 
so  on)  In  complex  equipment  Is  often  Incomplete,  vague, 
or  Incorrect.  As  long  as  this  Is  so,  any  series  of 
checks  and  test  readings,  though  apparently  well 
motivated  and  accomplished,  cannot  'close  In'  logically 
on  a  faulty  unit  (pp.5-6). 

The  premise  of  the  present  program  of  research  was  that  training  programs 
are  less  successful  than  they  could  be  because  they  fall  to  provide 
troubleshooters  with  the  knowledge  required  to  develop  a  sufficiently  rich 
conceptual  structure  of  the  equipment  they  are  working  with.  Accordingly, 
In  this  research  we  studied  the  nature  of  the  knowledge  base  that  Is 
necessary  to  repair  and  maintain  complex  electronic  devices. 


This  chapter  presents  a  rationale  for  the  wc-k  that  was  undertaken.  Its 
purpose  Is  to  delineate  our  underlying  theoretical  framework  for  the 
research.  Section  2.2  discusses  the  knowledge  base  required  for  complex 
problem  solving.  Section  2.3  discusses  the  nature  of  the  mental 
representation  of  complex  equipment  In  relation  to  contemporary  theories  of 
long-term  memory  and  the  representation  of  spatial  Information.  Section 
2.4  discusses  relevant  research  pertaining  to  the  nature  of  the  mental 
representation  possessed  by  novice  and  expert  problem  solvers.  Section  2.5 
present!  the  theoretical  perspective  that  we  adopted  for  the  conduct  of  the 
research. 


2.2  Knowledge  Base  and  Complex  Problem  Solving 

Troubleshooting  entails  the  Isolation  and  repair  of  malfunctioning 
components  In  a  device  and,  as  such.  Is  a  form  of  problem  solving.  It  may 
be  analyzed  In  terms  of  problem  solving  theories  (see,  e.g.,  Greeno,  1978, 
for  a  review).  However,  we  did  not  focus  exclusively  on  this  aspect  of  the 
troubleshooting  question.  It  was  presumed  that  problem-solving  procedures 
operate  on  a  substrate  of  knowledge,  which  Includes  a  mental 
representation,  of  a  device.  This  underlying  structure  must  be  clarified 
before  problem-solving  theories  can  be  applied. 

Although  much  attention  has  been  devoted  to  the  study  of  the  procedure 
people  use  In  problem-solving,  less  attention  has  been  paid  to  the  study  of 
the  knowledge  substrate  required  to  solve  complex  problems.  For  example, 
the  commonly  studied  problems  (Tower  of  Hanoi,  Missionaries  and  Cannibals, 
etc.)  require  a  very  limited  Information  base.  We  felt  that.  In  part,  what 
characterizes  good  problem  solvers  from  poor  ones  resides  In  the  knowledge 
base  they  bring  to  problem  solving  tasks  rather  than  mere  differences  In 
their  problem  solving  strategies. 


While  this  position  may  seem  to  be  uncontroverslal,  It  Is  Instructive  to 
note  that  much  previous  research  has  been  performed  that  presupposes  a  very 
Improvershed  mental  representation  on  the  part  of  the  problem  solver.  For 
example,  troubleshooting  Is  sometimes  viewed  as  a  decision  making  activity 
where  the  troubleshooter's  task  Is  to  Iteratively  select  and  test  system 
components  to  determine  whether  they  are  faulty  (e.g.,  Freedy  &  Crooks, 
1975;  Rouse,  1978).  From  this  perspective,  the  troubleshooter  must  decide 
which  system  component,  among  many,  to  test  on  a  given  Iteration.  Some 
have  suggested  (e.g..  Bond  &  Towne,  1979;  Rouse,  1978)  that  an  optimal 
strategy  to  use  Is  a  procedure  that  eliminates  half  the  number  of 
components  on  each  test  cycle.  To  be  Implemented,  a  strategy  such  as  this 
one  merely  requires  the  troubleshooter  to  have  a  list  of  system  components 
In  his  mind.  Unfortunately,  It  Is  not  clear  what  should  be  on  this  list, 
nor  how  It  should  be  organized.  It  Is  here  that  the  real  novice-expert 
differences  lie. 

We  believed  that  to  successfully  repair  complex  equipment,  a  troubleshooter 
requires  several  types  of  knowledge.  First,  factual  knowledge  of  the 
equipment  Is  required.  For  example,  troubleshooters  must  know  that  most 
electronic  devices  have  a  power  supply.  Second,  procedural  knowledge  Is 
needed.  Sometimes,  such  knowledge  Is  quite  specific;  for  example,  "To 
determine  whether  a  capacitor  is  faulty,  take  a  reading  of  the  voltage  at 
point  x.  If  the  voltage  departs  significantly  from  value  y,  the  capacitor 
Is  faulty."  Other  procedural  knowledge  Is  much  more  general,  as  In  a  set 
of  procedures  for  Identifying  what  may  be  wrong  when  no  power  Is  delivered 
to  a  device.  Third,  the  troubleshooter  requires  a  mental  representation  of 
the  faulty  device,  of  Its  components  and  their  Inter-relationships  to  each 
other.  The  basis  for  this  representational  knowledge  Is  provided  by 
schematic  drawings  of  the  equipment.  However,  the  availability  of 
schematic  drawings  does  not  mean  that  the  troubleshooter  understands  the 
device,  nor  that  he  can  generate  an  accurate  mental  representation. 


Nature  of  the  Mental  Representation 

It  Is  apparent  that  an  expert's  mental  representation  of  a  complex  device 
Is  not  Isomorphic  to  a  schematic  drawing  of  the  device.  In  fact.  It 
contains  much  more  Information.  (See  Brown,  Collins,  &  Harris,  1978  for  a 
similar  point  of  view.)  For  example,  an  expert  readily  can  Identify 
functional  units  within  a  circuit  that  are  not  directly  present  In  the 
diagram.  So  the  first  step  was  to  model  the  latent  knowledge  and  the 
cognitive  mechanisms  that  allow  the  expert  to  develop  an  enriched  mental 
representation  from  a  schematic  drawing. 

Contemporary  psychological  theorizing  provides  several  approaches  to  this 
problem.  One  approach  Is  to  view  the  mental  representation  of  spatial 
Information  In  terms  of  connected  networks  (Anderson,  1976;  Anderson  & 
Bower,  1973,  Collins  &  Loftus,  1975;  Norman  &  Rumelhart,  1975;  Schank, 
1972).  According  to  such  a  structural  model.  Individual  circuit  components 
are  represented  by  nodes  In  the  network  and  the  Interrelationships  among 
components  form  links  between  the  nodes.  Although  representations  of  this 
sort  can  be  constructured  to  mirror  closely  the  circuit  diagrams,  they  also 
permit  the  use  of  abstract  nodes  to  represent  the  hierarchical  relationship 
among  functional  groupings  of  components.  Host  of  the  work  with  these 
models  has  been  done  In  building  and  testing  theories  of  the  organization 
of  long-term  memory  and  thus  has  been  outside  the  context  of  problem¬ 
solving  tasks.  An  exception  Is  the  work  of  Bhaskar  and  Simon  (1977)  who 
have  undertaken  an  analysis  of  the  structure  of  long-term  memory  used  by 
students  solving  problems  In  a  college-level  course  In  chemical -engineering 
thermodynamics. 

A  second  approach  Is  to  view  the  Internal  representation  as  a  second-order 
Isomorphism  between  an  external  object  and  some  corresponding 
representational  process  with  the  brain  (Shepard,  1978).  According  to  this 
view,  the  relations  among  Imagined  objects  mirror  to  some  extent  the 
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functional  relations  among  the  same  objects  as  actually  perceived.  Studies 
based  on  this  view  (e.g.,  Cooper,  1975;  Kosslyn,  1975;  Shepard,  1978)  have 
emphasized  the  close  relationship  between  the  physical  nature  of  material 
and  the  chronometrlc  properties  of  the  response.  Because  of  the  spatial 
nature  of  circuit  diagrams.  Isomorphic  representation  such  as  these  play  an 
Important  role  In  any  model. 

A  third  approach  to  the  representation  of  knowledge  Is  procedural.  In  such 
a  view,  what  Is  known  Is  not  the  static  properties  of  a  circuit  diagram  but 
ways  of  operating  on  It.  A  worker  may  know  how  to  modify  a  power  supply  to 
better  filter  Its  output,  with  this  Information  taking  the  form  of 
procedures  for  altering  the  circuit  rather  than  a  compendium  of  facts  about 
supply  filters.  Because  of  Its  active  representation,  procedural  knowledge 
has  found  most  use  In  simulation  models  (e.g.,  Wlnograd,  1972)  and  problem 
solving  (Newell  &  slmon,  1972).  Some  models  (e.g.,  Anderson,  1976)  have 
Incorporated  both  procedural  and  other  forms  of  representation. 

deKleer  (1979)  and  deKleer  and  Brown  (1980)  have  described,  from  an 
artificial  Intelligence  point  of  view,  some  of  the  procedural  strategies 
that  are  required  to  analyze  the  operation  of  a  circuit.  In  particular, 
they  emphasize  the  need  for  multiple  procedural  strategies.  For  example, 
deKleer  (1979)  hypothesizes  that  people  use  topological,  functional,  and 
geometric  representations.  In  topological  analysis,  the  topology  of  a  new 
circuit  Is  compared  to  that  of  previously  recognized  circuits;  In 
functional  analysis,  the  behavior  of  the  overall  circuit  Is  determined  by 
combining  the  behavior  of  the  Individual  components;  and  geometric  analysis 
relies  on  the  tacit  graphical  language  engineers  use  when  they  describe 
circuit  topologies  on  paper.  These  representations  are  used  to  analyze 
circuits  In  terms  of  Its  "teleology."  Similarly,  Stevens  and  Collins 
(1980)  argue  that  people  maintain  multiple  representations  of  physical 
systems  such  as  of  the  rainfall  process. 


2.4  Comparing  Novices  to  Experts 

The  previous  research  most  relevant  to  understanding  how  people  represent 
complex  equipment  from  schematic  drawings  comes  from  research  comparing  the 
performance  of  experts  to  that  of  novices.  Most  research  In  this  area 
Indicates  that  experts  differ  from  novices  more  In  perceptual,  memorial 
abilities  than  In  logical,  problem-solving  abilities.  If  It  were  simply 
the  case  that  experts  know  more  about  the  task  than  the  novices,  there 
would  be  little  to  be  gained  from  constructing  an  elaborate  representation 
of  the  task.  However,  this  seems  not  to  be  so.  Experts  seem  to  have 
representations  that  differ  qualitatively  from  those  of  novices.  Studies 
supporting  this  position  are  becoming  common. 

Work  by  deGroot  (1966),  Chase  and  Simon  (1973),  and  Simon  and  Chase  (1973), 
comparing  the  performance  of  Master  and  weaker  chess  players.  Indicates 
that  Masters  do  not  "see"  ahead  further  than  the  weaker  players.  Instead, 
the  Masters  are  superior  to  weaker  players  In  their  ability  to  perform 
tasks  Involving  the  recall  of  actual  chess  positions.  The  superior 
performance  of  Masters  In  these  tasks  cannot  be  attributed  to  a  generally 
superior  visual  short-term  memory  capacity  of  the  Masters  because  when 
chess  pieces  are  placed  randomly  on  the  board,  recall  Is  equally  poor  for 
Masters  and  weak  players. 

Egan  and  Schwartz  (1979)  demonstrated  that  expert  electronic  trouble¬ 
shooters  have  a  richer  mental  representation  of  circuit  information  than 
novice  troubleshooters.  For  example,  expert  electronic  troubleshooters  are 
better  able  to  remember  circuit  diagram  information  than  are  novice 
troubleshooters,  and  in  reconstruction,  the  experts  recalled  the  diagrams 
In  groupings  of  functional  units.  The  skilled  technician's  advantage  In 
this  task  did  not  hold  for  non-meanlngfully  arranged  symbols. 
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Badre  (1979)  has  found  similar  results  for  the  recall  of  battlefield 
situation  displays.  Military  experts  show  a  marked  advantage  over  novices 
for  plausible  situation  displays  but  not  for  randomly  arranged  positions. 
Furthermore,  military  experts  recall  battlefield  units  on  the  basis  of 
their  functional  relationship  to  each  other. 

These  results  hold  for  other  spatial  tasks  such  as  the  recall  of  GO 
positions  (Reltman,  1976)  and  have  also  been  extended  to  other  non-spatlal 
tasks  such  as  for  the  recall  of  computer  programs  (Reltman,  McKelthen, 
Reuter,  &  Hlrtle,  1979)  and  the  solution  of  physics  problems  (Chi, 
Feltovlch,  &  Glaser,  1979;  Larkin,  1979;  Larkin,  McDermott,  Simon,  &  Simon 
1980) . 

The  most  Influential  theoretical  explanation  of  these  data  Is  that  experts 
perceive  spatial  stimuli  by  coding  the  stimuli  Into  groups  consisting  of 
several  elements  or  chunks.  In  one  version  of  this  theory  (Simon  &  Gil - 
martin,  1973),  the  chunks  have  verbal  labels  that  can  be  retained  in 
short-term  memory  and  decoded  at  the  time  of  recall.  It  Is  argued  that 
experts  quickly  represent  an  entire  spatial  representation  In  a  relatively 
small  number  of  chunk  labels  and  that  these  labels  can  be  used  to 
reconstruct  the  spatial  representation.  Pauses  between  successively 
recalled  elements,  the  estimated  size  and  number  of  chunks,  and  the 
correspondence  of  recall  groupings  in  copying  tasks  support  this 
hypothesis. 

The  data  are  In  good  agreement  with  the  semi-hierarchical  theory  of  the 
mental  representation  of  complex  equipment  discussed  in  the  following 
section  (Indeed,  they  were  part  of  the  motivation  for  our  model). 
Moreover,  our  theory  provides  a  mechanism  to  account  for  Egan  and 
Schwartz's  (1969)  observation  that  experts  can  quickly  label  a  circuit 
diagram  as  belonging  to  a  given  class  (e.g.,  "some  sort  of  power  supply"). 


In  sum,  we  do  not  feel  that  the  understanding  of  skilled  performance  In 
tasks  such  as  circuit  analysis  Is  possible  without  considering  multiple 
domains  of  knowledge.  All  are  necessary  to  explain  the  richness  of  an 
expert  troubleshooter's  mental  constructs,  and  our  ultimate  model 
Incorporates  elements  of  each.  In  particular,  we  believe  that  the 
difference  between  experts  and  novices  lies  as  much  or  more  In  the  ability 
of  experts  to  draw  from  a  larger  collection  of  operations  —  or  procedural 
knowledge  —  than  In  an  understanding  of  the  structural  and  functional 
nature  of  the  parts  of  the  task  or  In  the  reference  to  prototypes.  Much 
previous  work  has  Ignored  the  dynamic,  procedural  aspects  in  favor  of  more 
static  conceptions  of  the  tasks.  The  following  section  presents  our 
theoretical  position  In  more  detail. 

2.5  Theoretical  Perspective 

As  one  reviews  the  literature  on  the  mental  representation  of  tasks  or 
stimuli,  one  Is  struck  with  the  extent  to  which  the  derived  representations 
are  well  matched  to  the  task.  This  suggested  to  us  that  the  subject  Is 
able  to  adopt  a  mental  representation  that  is  closely  concordant  with  local 
processing  demands.  The  true  mental  representation  must  have  latent  in  it 
a  variety  of  possible  forms  and  structures.  This  observation  dictated  one 
fundamental  principle  behind  the  experiments  that  we  present  here:  any 
analysis  of  the  mental  characteristic  of  a  domain  of  knowledge  must  derive 
from  a  variety  of  tasks  and  must  posit  a  variety  of  Individual 
representations. 


We  started  by  making  a  distinction  between  two  ways  of  classifying  the 
content  of  a  knowledge  structure.  The  first  refers  to  the  substantive 
content  of  the  representation,  to  what  facts  It  describes;  the  second  to 
the  way  In  which  the  Information  relates  to  the  subjects'  knowledge  base. 
Both  of  these  viewpoints  were  further  subdivided  as  Is  discussed  below.  We 
emphasize  that  this  Is  not  a  dichotomy  In  the  knowledge  Itself  —  a 


particular  piece  of  Information  does  not  belong  to  one  or  the  other  —  but 
rather  two  aspects  of  the  same  mental  structure,  both  of  which  must  be 
treated  to  fully  comprehend  the  Information  processing  abilities  of  a 
subject. 

Content.  The  content  of  the  mental  representation  Is  composed  of  a  number 
of  Interrelated  and  overlapping  structures.  These  structures  may  be 
hierarchical  In  form  (although  this  Is  not  crucial).  The  hierarchical 
character  arises  from  the  tendency  to  view  portions  of  the  device  as  units, 
without  examining  their  fine  structure  unless  necessary.  For  example,  one 
may  think  of  a  logic-circuit  component  as  a  flip-flop,  without  analyzing  it 
further  unless  forced  to  do  so.  This  hierarchical  tendency  has  been 
supported  In  modern  Integrated  circuits  by  the  physical  modularization  of 
rather  complex  functions  in  single  chips  or  modules;  In  computers,  for 
example,  CPUs,  Interface  units  (e.g.,  UARTs),  and  fairly  extensive  memory 
drivers  may  appear  as  single  units. 

The  overlapping  nature  of  the  representation  comes  from  the  fact  that  a 
particular  component  participates  In  several  organizations  at  once.  In 
some  computer  designs,  for  example,  the  circuit  representing  the  fourth  bit 
of  an  accumulator  may  logically  be  analyzed  as  part  of  that  accumulator,  or 
may  be  considered  as  part  of  the  array  of  fourth  bits  over  a  series  of 
registers  such  as  the  accumulator,  a  program  converter,  etc.  (It  Is 
Interesting  that  both  forms  of  organization  are  reflected  In  the  physical 
design  of  different  computers.)  As  a  more  prosaic  example,  an  electric 
fuel  pump  In  an  automobile  participates  In  both  the  electrical  and  the  fuel 
system  of  the  car.  Presumably  all  of  these  overlapping  organizations  are 
accessible  to  the  subject  who  Is  employed  at  a  particular  time  depending  on 
the  task.  The  processing  demands  that  the  subject  places  on  the  mental 
representation  enables  one  form  or  the  other. 


We  feel  that  the  cognitive  representation  of  information  given  In  a  circuit 
diagram  may  be  usefully  described  as  a  set  of  parallel  networks.  In  part, 
these  networks  have  a  hierarchical  structure.  In  that  the  terminal  nodes 
represent  Individual  circuit  components  (resistors,  capacitors,  etc.);  the 
Intermediate  nodes  represent  either  functional  units  (rectifiers, 
amplifiers,  etc.),  or  physically  proximal  collections  of  components,  and 
the  highest  nodes  represent  the  total  circuit  (a  power  supply,  an  Inverter, 
etc.). 

Nature  of  the  Knowledge.  The  second  scheme  of  organization  relates  to  the 
way  that  the  Information  Is  used.  Different  tasks  use  knowledge  In 
different  ways.  Although  the  range  of  possible  uses  probably  forms  a 
complex  and  multidimensional  space,  a  good  case  can  be  made  for  classifying 
knowledge  about  a  complicated  mechanism  (such  as  a  circuit  diagram)  Into 
three  general  classes: 

(1)  Structural  and  functional  knowledge  deals  with  the  way  that  a 
device  Is  constructed  and  the  role  that  Its  parts  play  In 
operation.  One  may  know,  for  example,  that  a  transformer 
serves  to  change  the  voltage  of  an  AC  supply,  that  a 
particular  combination  of  transltors  acts  as  a  flip-flop,  and 
so  forth.  Fundamentally,  this  knowledge  Is  static;  It 
describes  the  way  that  the  device  works. 

(2)  Prototypical  Knowledge  relates  one  device  to  more  general 
prototypes.  Devices  are  not  understood  In  Isolation,  but  are 
related  to  other  devices.  For  example,  experienced 
technicians  are  able  to  quickly  recognize  that  several 
difficult  circuit  diagrams  represent  the  same  class  of  device. 
This  suggests  a  set  of  procedures  that  force  the  constituent 
elements  of  a  circuit  diagram  Into  prototypical 
configurations.  One  portion  of  a  circuit  is  a  rectifier. 


another  a  Schmitt  trigger,  etc.  Two  processes  seem  to  be 
Involved.  First,  some  form  of  bottom-up  mechanism  simplifies 
the  representation  of  the  diagram  by  replacing  groups  of  nodes 
with  a  single  node.  Second,  a  top-down  mechanism  attempts  to 
fill  In  missing  nodes  In  partially  matched  prototypes.  This 
knowledge  Is  rarely  in  the  form  of  an  exact  parallel  between 
real  devices,  but  relates  the  operation  of  any  device  to  an 
abstracted  prototype.  This  prototypical  knowledge  reduces 
greatly  the  burden  of  the  structural  and  functional  facts. 

(3)  Procedural  knowledge  gives  ways  to  manipulate  the  device. 
Many  tasks  require  more  than  a  static  comprehension  of  a 
device,  demanding  that  some  modification  be  made  to  some 
operation  performed  on  It.  The  knowledge  necessary  to  do  this 
is  different  from  the  other  two  types,  embodying  a  series  of 
procedures  (algorithmic  or  heuristic)  for  changing  the  device. 
Procedural  knowledge  Is  the  most  complex  of  the  three,  and 
draws  heavily  on  the  others. 

We  feel  that  training  programs  fall  because  the  mental  representation  is 
unobservable,  and  consequently  It  Is  difficult  to  determine  whether  the 
troubleshooter  has  developed  an  adequate  representation.  Moreover,  It  Is 
not  clear  how  such  a  representation  Is  attained.  However,  we  do  know  that 
with  experience,  troubleshooters  become  more  proficient  at  their  job. 
Presumably,  increased  proficiency  derived  through  experience  Is  due  to  a 
more  developed  mental  representation  knowledge  base. 

Our  goal  was  to  obtain  a  better  understanding  of  the  knowledge  base  that 
skilled  troubleshooters  bring  to  bear  in  complex  equipment.  Of  the  three 
types  of  knowledge  that  we  have  discussed,  procedural  knowledge  is  the  most 
difficult  to  study,  for  It  Is  the  most  abstract,  and  depends  on  the 


structural /functional  knowledge  for  Its  operation.  But  it  Is  also  the  most 
Important,  because  this  knowledge  Is  valuable  only  to  the  extent  that  it 
lets  Its  possessor  manipulate  equipment.  Furthermore,  preliminary  work  on 
structural /functional  knowledge  provides  the  necessary  background  to 
understand  procedural  knowledge. 

We  note  that  at  this  point  we  have  described  a  general  theoretical 
position,  but  not  the  specific  models  that  Instantiate  It  (for  this 
distinction,  see  Wlckens,  1982,  Chapter  I).  Specific  models  for  these 
processes  could  be  formulated  using  a  number  of  conventional 
representations  such  as  production  systems. 


3.  REVIEW  OF  FIRST-YEAR  WORK: 

INVESTIGATION  OF  STRUCTURAL/FUNCTIONAL  KNOWLEDGE 

3.1  Overview 

Reviews  of  the  literature  on  the  mental  representation  of  tasks  or  stimuli 
reveal  that  the  derived  representations  are  surprisingly  well  matched  to 
the  task.  This  suggests  that  a  person  Is  able  to  adopt  a  mental 
representation  that  is  closely  concordant  with  local  processing  demands. 
The  true  mental  representation  must  have  a  variety  of  possible  forms  and 
structures  that  are  latent  in  that  representation.  Consequently,  any 
analysis  of  the  mental  characteristics  of  a  domain  of  knowledge  must  derive 
from  a  variety  of  tasks  and  must  posit  a  variety  of  Individual 
representations.  Also,  the  understanding  of  skilled  performance  in  tasks 
such  as  circuit  analysis  requires  consideration  of  multiple  domains  of 
knowledge.  That  Is,  different  domains  are  necessary  to  explain  the 

richness  of  an  expert  troubleshooter's  mental  constructs,  and  a 
comprehensive  model  must  Incorporate  elements  of  each  domain. 

The  experiment  conducted  during  the  first  year  was  designed  to  assess  the 
structural  and  functional  knowledge  of  electronic  devices  possessed  by 
technicians  varying  In  skill  level.  This  work  provided  the  necessary 
background  for  the  Investigation  of  procedural  knowledge  as  related  to 
electronic  trouble-shooting.  The  experiment  performed  was  a  composite  of  a 
circuit-reconstruction  task  and  a  circuit-partitioning  task. 

The  principal  finding  evident  from  the  data  was  the  large  degree  of  varia¬ 
bility  In  the  performance  among  the  subjects  on  the  reconstruction  and 
cluster-generating  tasks,  and  the  extent  of  overlap  between  the  different 
ability  groups  on  those  tasks.  These  data  stand  In  contrast  to  the 
relatively  large  differences  between  skill  levels  that  appeared  In  the  time 
and  error  measures  with  respect  to  overall  performance.  Even  If  a  large 


sample  could  provide  statistically  slgni*  differences  between  skill 
levels,  the  size  of  the  effect  as  a  pro.^f'L.on  of  variability  would 
probably  not  be  very  great.  Thus,  the  investigation  of  structural /func¬ 
tional  knowledge  was  not  the  most  productive  place  for  further  research 
effort. 

Identifiable  differences  In  the  organization  of  a  diagram  during  learning 
and  recall  do  not  appear  to  be  the  most  sensitive  loci  of  proficiency 
differences.  For  reasonably  simple  and  well-learned  material,  the  physical 
properties  of  the  diagram  layout  may  dominate  performance.  It  Is 
conceivable  that  differences  could  appear  between  subjects  of  different 
proficiency  levels,  with  the  less  proficient  subjects  being  more  bound  by 
the  physical  construction  of  the  target  circuit,  and  the  more  proficient 
subjects  being  more  bound  by  the  logical  construction.  In  conventionally 
well-drawn  circuit  diagrams,  howe'er,  these  two  organizations  coincide, 
minimizing  differences  in  performance. 

In  contrast,  more  substantial  differences  are  more  likely  to  appear  In  the 
way  In  which  th«  circuit  diagrams  are  manipulated,  that  Is,  In  operations 
tnat  ary  perfonaed  on  the  diagrams.  Even  with  Improperly  drawn  diagrams, 
where  the  logical  and  physical  aspects  conflict,  the  expert's  advantage 
would  come  through  an  ability  to  reorganize  the  circuit.  Thus,  the 
principal  performance  differences  between  the  most  skilled  and  least 
skilled  subjects  may  be  a  result  of  differences  In  their  respective  levels 
of  procedural  knowledge.  For  this  reason,  the  second  year  of  research  work 
was  designed  to  examine  performance  on  tasks  that  require  technicians  to 
manipulate  circuits  in  prescribed  ways.  Such  research,  focusing  on  the 
Investigation  of  procedural  knowledge,  complements  the  exploration  of 
Individual  differences  In  structural /functional  knowledge  performed  during 
the  first  year  of  work. 


4.  REVIEW  OF  SECOND-YEAR  WORK: 
INVESTIGATION  OF  PROCEDURAL  KNOWLEDGE 


4.1  Overview 


Knowing  the  way  In  which  the  mental  representation  of  experts  differs  from 
that  of  novices  does  not  directly  Indicate  how  the  experts'  representation 
leads  to  superior  troubleshooting.  Greeno  (1978)  makes  the  same  point  by 
noting  that  while  current  theorizing  gives  an  explanation  of  the  skill  that 
chess  masters  show  In  the  short-term  recall  of  positions,  there  has  been  no 
strong  theoretical  analysis  showing  how  the  existence  of  a  large  store  of 
recognizable  patterns  contributes  to  successful  problem  solving.  Thus, 
tasks  that  tap  procedural  knowledge  are  Intended  to  create  a  bridge  between 
the  structural /functional  nature  of  the  mental  representation,  and  the 
procedures  acting  on  this  representation  that  enable  problem-solving  and 
troubleshooting  activities. 

Three  tasks  were  designed  to  study  procedural  knowledge.  Each  task 
required  the  subject  to  manipulate  a  circuit  diagram  In  some  manner  such 
that  differences  In  algorithmic  or  heuristic  procedures  for  changing  the 
device  could  be  evaluated  as  a  function  of  skill  level.  Our  Interest  here 
was  primarily  In  the  Interaction  of  proficiency  with  the  specific  diagram 
manipulation.  Task  I  required  the  subject  to  locate  an  error  In  the 
construction  of  a  circuit  on  the  basis  of  symptoms;  Task  2  required  the 
subject  to  change  the  function  of  a  circuit;  and  Task  3  asked  the  subject 
to  complete  a  missing  segment  of  a  circuit.  Each  experiment  Is  described 
briefly  below. 


Task  1:  Error  Correction  ("FIX"  Task 


In  this  task,  technicians,  varying  In  skill  level,  were  presented  with  a 
diagram  containing  an  error  In  It  and  with  symptoms  of  Its  malfunction. 
The  subject's  task  was  to  find  the  error  and  to  correct  or  fix  It.  While 
performing  this  task,  a  record  was  maintained  of  the  operations  that  the 
subject  used  and  of  their  order.  In  this  way,  the  nature  and  order  of  the 
procedures  could  be  catalogued  and  compared  across  the  specified  stimulus 
material  as  a  function  of  subject  skill  level. 

4.3  Task  2:  Chanoe  Function  I "ALTER"  Task 


A  good  Indication  of  a  subject's  deep-level  comprehension  of  a  circuit  is 
provided  by  the  readiness  with  which  that  circuit  can  be  modified.  In  this 
experiment,  subjects  were  presented  with  a  circuit  diagram  of  a  device  that 
was  complete  and  would  operate  as  described.  The  subject's  task  was  to 
make  a  specific  change  In  one  of  :he  circuit's  parameters.  For  example.  In 
the  small  diagram  used  In  the  first  year  of  work,  the  subject  might  be 
asked  to  change  the  frequency  of  the  tuning  unit.  As  In  Task  1,  protocols 
of  the  subject's  operations  were  catalogued  and  compared. 

4.4  Task  3:  Complete  Circuit  ("COMPLETE"  Task 


This  task  required  the  subject  to  fill  in  a  missing  segment  of  a  circuit 
diagram  so  that  the  circuit  would  perform  a  particular  function.  Circuits 
were  constructed  In  which  a  particular  segment  of  the  diagram  was  left 
blank.  These  omissions  subsumed  a  functional  unit  of  the  circuit  (not  just 
a  single  component),  but  were  not  so  large  as  to  require  substantial 
redesign.  As  In  Tasks  1  and  2,  a  record  was  maintained  of  the  procedural 

steps  used  by  the  subjects  to  perform  the  task.  These  protocols  were  then 
summarized  and  compared  between  subject  groups  of  different  skill  levels. 


The  sample  of  12  novice  technicians  for  the  second  year  of  work  were  drawn 
from  electronics  trade  schools  In  the  Los  Angeles  area  that  offer  a 
two-year  training  program  In  electronics  to  high  school  graduates.  The  9 
expert  technicians  were  currently  employed  with  2  to  4  years  of  college 
education  and  an  average  of  5  years  of  work  experience  in  electronics. 

4.6  Materials 

Six  circuits  were  designed  for  use  In  the  studies  of  procedural  knowledge. 
Three  of  these  are  analog  circuits  with  a  level  of  complexity  comparable  to 
that  of  the  medium-sized  circuit  used  In  the  first  year's  work.  The 
remaining  three  circuits  were  drawn  from  digital  materials  because  (a)  much 
of  contemporary  electronic  equipment  Is  digital,  (b)  much  of  the  current 
technical  training  of  technicians  Is  focused  on  digital  circuits,  and  (c) 
we  wished  to  study  procedural  knowledge  as  It  relates  to  both  types  of 
circuits. 

Three  versions  of  each  circuit  were  prepared:  a  complete  working  circuit 
for  the  alter  function  task;  an  Incomplete  circuit  for  the  circuit 
completion  task;  and  a  modified,  non-working  circuit  for  the  error 
correction  task.  These  circuits  are  presented  in  Figures  4-1A  to  4-6C.  In 
each  figure,  the  complete,  working  version  Is  shown  In  panel  A,  the 
Incomplete  version  In  panel  B,  and  the  modified,  non-working  version  In 
panel  C. 

4.7  General  Procedure 

Three  tasks  were  used  to  study  procedural  knowledge.  Each  task  required 
subjects  to  manipulate  a  circuit  diagram  In  some  manner  such  that 
differences  In  algorithmic  or  heuristic  procedures  for  changing  the  device 
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FIGURE  4-2A. 

VOLTAGE  TO  FREQUENCY  CONVERTER  (ANALOG):  COMPLETE,  WORKING  CIRCUIT 
[FOR  "ALTER"  TASK] 


FIGURE  4-2B. 

VOLTAGE  TO  FREQUENCY  CONVERTER  (ANALOG):  INCOMPLETE  CIRCUIT 
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FIGURE  4-2C. 

VOLTAGE  TO  FREQUENCY  CONVERTER  (ANALOG):  MODIFIED,  NON-WORKING  CIRCUIT 
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PREAMPLIFIER  -  ONE  CHANNEL  (ANALOG):  INCOMPLETE  CIRCUIT 
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FIGURE  4-4A. 
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UNIVERSAL  4-BIT  BINARY  COUNTER  (DIGITAL):  MODIFIED,  NON-WORKING  CIRCUIT 
fFOR  "FIX"  TASK1 


TO  DIGITAL  CONVERTER  (DIGITAL):  MODIFIED,  NON-WORKING  CIRCUIT 
FIX"  TASK] 


could  be  evaluated  as  a  function  of  the  subject's  skill  level.  Our 
Interest  here  was  primarily  In  the  Interaction  of  subject  proficiency  with 
the  specific  circuit  manipulation  required  In  the  diagram. 


The  first  task  required  the  subject  to  locate  and  correct  an  error  In  the 
construction  of  a  circuit  on  the  basis  of  symptoms;  the  second  task 
required  the  subject  to  change  the  function  of  a  circuit;  and  the  third 
task  asked  the  subject  to  complete  a  missing  segment  of  a  circuit.  Each 
subject  solved  problems  of  all  three  types  over  the  course  of  one  two-hour 
session.  Each  task  was  performed  twice,  once  with  a  digital  circuit  and 
once  with  an  analog  circuit.  The  selection  of  circuits  for  tasks  was 
counterbalanced  across  subjects  within  each  skill  level;  and  the  order  of 
presentation  of  the  three  different  tasks  was  also  counterbalanced. 


Considerable  effort  was  devoted  to  training  the  subjects  to  produce  the 
verbal  protocols  while  solving  the  task  problems.  The  following 
Instructions  were  read  to  the  subjects; 


"We  are  Interested  In  your  solutions  to  the  problems,  but  we 
are  also  Interested  In  how  you  arrive  at  your  solutions. 
There  may  be  several  different  ways  to  solve  the  problems  and 
there  are  certainly  several  different  ways  to  arrive  at  any 
one  solution.  We  would  like  to  keep  a  record  of  everything 
you  do  in  coming  up  with  your  solutions.  One  way  that  we  will 
do  this  Is  to  have  you  think  out  loud  as  you  are  solving  the 
problems.  Since  most  people  do  not  naturally  think  out  loud, 
this  may  require  some  concentration  on  your  part. 

First,  It  would  be  most  useful  to  us  if  we  could  write  down  a 
list  of  distinct  steps  that  you  go  through  to  arrive  at  your 
final  solution.  To  help  us  do  this,  we  want  you  to  tell  us 
what  you  are  thinking  at  each  point  as  you  work  toward  your 
solutions,  and  also  wh,y  you  are  thinking  about  the  steps  you 
give  us. 

It  Is  crucial  to  us  that  you  do  not,  for  any  reason,  edit 
anything  out  from  your  thinking.  For  example.  If  you  see  a 
step  that  you  would  like  to  carry  out  but  you  don't  see  how  to 
do  It  right  away,  please  describe  what  you  would  like  to  do 


anyway  and  why  you  would  like  to  do  It.  Also,  If  you  start  to 
do  something  and  then  decide  against  It,  tell  us  about  It 
anyway  and  why  you  changed  your  mind.  Do  not  edit  anything 
out.  We  are  not  only  Interested  In  the  steps  that  lead 

3Trectly  to  your  solution,  but  we  are  also  very  Interested  In 
the  things  you  think  about  and  then  decide  are  wrong.  In 
short,  we  need  a  record  of  everything  you  think  about  until 
you  finally  come  up  with  your  solution. 

To  make  a  record  of  your  thoughts,  I  will  take  notes  on  what 
you  say.  Sometimes  I  will  have  to  ask  you  what  you  mean.  As 
a  backup,  I  will  also  tape  record  what  you  say  so  that  I  can 
go  back  later  and  fill  In  things  that  I  may  have  missed. 

To  reiterate,  we  are  not  so  much  Interested  In  finding 
solutions  to  the  problems.  We  already  know  a  few  ways  that 
the  problems  can  be  solved.  What  we  are  primarily  Interested 
In  Is  an  ongoing  record  of  what  people  do  toward  arriving  at 
the  solutions.  We  are  just  as  much  Interested  In  things  that 
you  do  and  then  decide  are  wrong,  as  things  that  you  do  and 
decide  are  correct." 

Performance  on  each  of  the  three  tasks  was  preceded  by  an  example,  using  a 
very  simple  three-way  light  circuit.  For  the  FIX  task,  an  extra  connection 
was  added  that  would  burn  out  the  battery;  for  the  ALTER  task,  the  subject 
was  shown  how  to  add  a  third  switch  to  turn  the  light  on  or  off;  for  the 
COMPLETION  task,  connections  were  deleted. 


Each  subject's  verbalizations  during  the  problem-solution  periods  were  tape 
recorded  and  transcribed  for  later  analysis.  The  experimenter  also 
recorded  on  paper  supplementary  notes  concerning  the  subject's  operations. 
To  Insure  a  hard-copy  record  of  the  operations,  the  subjects  were  asked  to 
draw  their  solutions  on  copies  of  the  circuit  layouts.  If  the  subject 
changed  his  or  her  mind  after  beginning  a  solution,  an  unmarked  copy  of  the 
diagram  was  provided.  This  was  done  to  maintain  a  complete  record  of  all 
the  operations  attempted. 


4.8 


Results 


4.8.1  Accuracy  of  Solutions.  Accuracy  data  in  terms  of  the  proportion  of 
problems  for  which  workable  solutions  were  provided  for  each  task  are 
presented  In  Table  4-1.  The  data  confirm  that  the  tasks  studied  here  were 
sufficiently  difficult  to  promote  large  accuracy  performance  differences 
with  respect  to  skill  level;  and,  clearly,  the  expert  subjects  did 
significantly  better  than  the  novice  subjects  on  all  tasks.  Furthermore, 
given  that  performance  by  experts  (at  least  on  the  FIX  and  COMPLETE  tasks) 
was  not  perfect.  It  Is  evident  that  the  tasks  required  procedural  knowledge 
and  were  not  Identifiable  "textbook11  examples  of  standard  problems. 
Nevertheless,  as  demonstrated  In  previous  stages  of  this  research, 
differences  In  accuracy  performance  are  not  particularly  useful  for 
understanding  procedural  knowledge. 

4.8.2  Classification  of  Protocols.  The  transcribed  protocols  first  were 
Inspected  for  patterns  by  an  experimenter  who  was  blind  to  the  skill-level 
classification  of  each  subject.  All  of  the  protocols  could  be  described  In 
a  general  respect  using  four  classifications  of  problem  solving  behavior: 
directed,  re-directed.  Immediate,  and  trial -and-error.  Directed  behavior 
is  characterized  by  an  apparent  plan  toward  solution,  with  goals  and 
If-then  operations.  This  procedure  Is  similar  to  what  Simon  (1969)  has 
called  a  means-end  analysis  where  the  task  Is  to  find  the  difference 
between  a  desired  state  and  the  existing  state,  and  then  to  find  the 
correlating  process  that  will  erase  the  difference.  Re-directed  behavior 
has  the  characteristics  of  directed  behavior,  plus  the  addition  of  at  least 
one  decision  on  the  part  of  the  subject  that  the  current  direction  is 
Incorrect  and  a  new  direction  Is  pursued.  This  behavior  Is  similar  to 
"hill-climbing"  (Atwood  &  Poison,  1976),  where  problem  solution  substates 
are  sometimes  nearer  to  the  goal  than  alternative  states,  but  they  are  off 
the  solution  path  because  of  "local  highs"  (e.g.,  the  Tower  of  Hanoi 
problem).  Immediate  solution  behavior  contains  no  sequence  of  operations. 


TABLE  4-1 


Proportion  Of  Problems  For  Which 
Workable  Solutions  Were  Provided 


FIX  ALTER  COMPLETE 


EXPERT  .50  1.00  .75 

NOVICE  .20  .20  .25 


but  rather  a  "snap"  solution  given  quickly.  This  would  Include  Instances 
of  "Insight"  (Wallas,  1926).  Trial -and-error  behavior  Is  best 
characterized  as  "maybe  I  would  try  X;  no  maybe  I  would  try  Y,"  and  perhaps 
Indicating  attempts  to  match  the  current  problem  with  "textbook"  solutions. 

This  behavioral  classification  was  carried  out  by  one  of  the  project 
Principal  Investigators  who  was  blind  as  to  which  protocols  were  from 
expert  versus  novice  technicians.  Each  protocol  was  classified 
systematically  based  on  clear  breaks  In  the  solution  paths.  If-then 
statements,  and  admissions  of  Incorrect  attempts,  and  guessing.  The 
results  are  presented  In  Table  4-2.  As  Is  obvious  from  the  data  pattern, 
the  expert  problem  solution  protocols  were  predominantly  classified  (56%) 
as  "directed"  behavior  —  the  highest  level  of  cognitive  performance.  And, 
much  of  the  balance  of  the  expert  protocols  (28%)  were  classified  as 
"re-directed"  behavior.  In  contrast  to  the  experts,  the  novices  showed  no 
predominant  behavioral  classification  and  only  33%  of  the  protocols  were 
classified  as  "directed,"  with  nearly  the  same  proportion  being  classified 
as  "trial -and-error." 

The  problem  solving  behavior  classifications  also  revealed  Interesting 
Interactions  between  skill  level  and  the  nature  of  the  task.  Table  4-3 
presents  those  data.  In  particular,  the  most  dramatic  difference  between 
skill  levels  occurs  with  the  ALTER  and  COMPLETE  task;  In  these  tasks,  75% 
of  the  protocols  generated  by  the  experts  were  classified  as  directed; 
whereas,  49%  of  the  protocols  generated  by  the  novices  were  classified  as 
trial -and-error.  The  ALTER  and  COMPLETE  tasks  are  task  least  likely  to 

have  received  prior  training  In  the  trade  schools.  It  Is  reasonable  that 
these  tasks  represent  exercises  where  the  novices  cannot  mimic  the 
systematic  behavior  of  experts,  and  therefore  these  type  of  data  may  prove 
most  diagnostic  In  evaluating  differences  between  experts  and  novices  on 
procedural  knowledge. 
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TABLE  4-4 

Average  Solution  Time  (Min) 


FIX 


ALTER 


COMPLETE 


4.8.3  Solution  Time.  Table  4-4  presents  the  average  problem  solution  time 
as  a  function  of  skill  level  and  task.  Most  striking  Is  the  lack  of 
differences  In  solution  time  between  skill  levels.  The  only  apparent 
difference  Is  In  the  COMPLETE  task,  where  the  experts  studied  the  diagram 
for  nearly  two  minutes  longer  before  Initiating  a  solution  sequence.  This 
Is  consistent  with  the  finding  that  novices  resort  to  trial -and-error 
behavior  on  the  completion  task. 

4.8.4  Content  Differences.  Several  reliable  differences  In  the  content  of 
the  protocols  as  a  function  of  skill  level  were  apparent  and  these 
Important  differences  formed  the  basis  for  procedural  guidelines  that  were 
established  and  evaluated  during  the  third  year  of  work. 

(1)  Experts  devoted  considerable  time  at  the  beginning  of  each 
problem  "learning  how  the  circuit  works"  before  Initiating  a 
problem-solution  sequence.  Specifically,  experts  partitioned 
the  circuit  while  labeling  key  parts  from  the  problem 
description  and  traced  the  flow  of  Information  through  the 
circuit  while  noting  the  activity  along  the  route.  The 
novices'  Initial  Inspection  time  was  spent  in  unsystematic 
viewing  of  the  circuit. 

(2)  The  second  major  difference  was  In  the  establishment  of  a  plan 


(3)  The  experts  proceeded  through  a  sequence  of  steps,  making 
changes,  additions,  or  deletions  In  the  circuit,  and  each  step 
was  followed  by  tracing  the  effects  of  the  action  taken  on  the 
functioning  of  the  circuit.  Novices  rarely  studied  the  Impact 
of  each  solution  step  separately;  and  on  occasion,  the  Impact 
of  the  entire  solution  was  not  traced  through  the  circuit. 

4.9  Conclusions 


On  all  three  tasks,  the  experts  were  more  accurate  with  their  solutions 
than  the  novices;  but  more  Importantly,  the  experts  were  classified  on  the 
basis  of  their  verbal  protocols  to  be  more  systematic,  orderly,  and 
directed  In  their  problem  solving  strategies.  Thus,  It  Is  plausible  that 
the  principal  performance  differences  between  the  most  skilled  and  least 
skilled  subjects  are  a  result  of  differences  in  their  respective  levels  of 
procedural  knowledge.  This  hypothesis  was  evaluated  In  the  third  year  of 
work.  In  which  the  protocol  differences  were  translated  into  guidelines 
toward  Improving  circuit  comprehension  and  troubleshooting  In  less  skilled 
technicians. 


5.  REVIEW  OF  THIRD  YEAR  WORK: 

DERIVATION  AND  VALIDATION  OF  PROCEDURAL  GUIDELINES 


5.1  Overview 

The  first  year  of  work  explored  a  characterization  of  the  knowledge 
structures  that  differentiate  technicians  of  different  skill  levels.  The 
second  year  of  work  Increased  our  understanding  of  the  procedural  knowledge 
that  technicians  apply  to  the  mental  representation  of  electronic  circuits 
to  perform  troubleshooting  and  problem  solving  tasks.  These  latter 
differences,  we  believe,  characterize  the  principal  differences  between 
skilled  and  unskilled  subjects.  The  results  were  then  validated  and 
applied. 

It  was  possible  to  combine  the  validation  of  our  findings  with  some  more 
practical  goals.  Based  on  the  differences  that  we  have  observed,  our  next 
task  was  to  develop  and  test  procedural  guidelines  for  comprehending 
circuits,  building  circuits,  and  troubleshooting.  Our  approach,  then, 
during  the  third  year  of  work  was  to  describe  how  expert  technicians 
approach  the  range  of  tasks  studied  during  the  second  year  of  work,  develop 
specific  procedural  guidelines  based  on  this  knowledge,  and  then  conduct  an 
experiment  to  validate  the  usefulness  of  the  guidelines  toward  Improving 
performance.  The  latter  step  was  accomplished  by  providing  one  group  of 
less  proficient  technicians  with  the  guidelines,  and  comparing  their 
performance  to  that  of  a  group  of  technicians  with  comparable  skills  who 
were  not  given  the  guidelines. 

This  general  procedure  has  been  employed  successfully  by  Samet  and 
Gelselman  (1981)  to  develop  guidelines  for  summarizing  tactical 
Intelligence  data.  The  basic  notion  Is  that  experts  have  a  mental 
representation  of  their  area  of  expertise  that  is  concordant  with  the 


Information  processing  requirements  that  operations  on  the  material  entail. 
This  representation  can  be  translated  to  some  extent  Into  guidelines  that 
can  be  used  to  Improve  the  performance  of  other  Individuals  who  have,  as 
yet,  a  more  limited  or  otherwise  less  coherent  mental  representation  of 
these  procedures. 

5.2  Procedural  Guidelines 

As  outlined  In  the  discussion  of  the  second  year's  experiments  on 
procedural  knowledge  (see  Section  4.8.4),  striking  differences  were 
observed  In  the  content  of  the  protocols  produced  by  the  experts  versus 
less-proficient  technicians.  Based  on  those  differences,  the  following 
guidelines  were  derived  for  use  In  the  validation  experiment.  The 
guidelines  apply  to  two  parts  of  the  problem  solving  sequence:  The 
preparatory,  circuit  understanding  phase  and  the  problem  solution  phase. 
The  guidelines  were  designed  to  ensure  that  subjects  would:  (a)  attempt  to 
understand  the  circuit  prior  to  Initiating  a  solution,  (b)  establish  a  plan 
of  attack,  (c)  proceed  through  a  sequence  of  goal-directed  steps,  and  (d) 
trace  the  effects  of  the  action. 

Part  1:  Understanding  the  circuit 

PURPOSE 


Start  with  the  purpose  of  the  circuit.  Does  It  have  an  Input?  An 
output?  Does  It  do  something  (for  Instance,  make  a  noise)?  Are 
there  controls  that  you  would  adjust  when  using  It?  Identify 
these  Inputs,  outputs,  and  controls. 


INFORMATION  FLOW 


Try  to  Identify  how  "Information"  flows  through  the  circuit.  How 
does  the  Input  get  to  the  output?  Where  Is  It  acted  on  by  the 
controls  mentioned  above.  Although  there  may  be  several  different 
paths  connecting  Input  to  output,  try  to  trace  one  main  path. 

THE  PARTS  OF  THE  CIRCUIT 

Try  to  understand  what  the  different  parts  of  the  circuit  do. 
What  are  the  various  components  and  how  do  they  act?  Two  things 
can  be  helpful  here: 

a.  Subgroups 

Do  certain  groups  of  components  go  together  and  act  as  a 
whole?  Several  parts  may  form  a  voltage  divider,  an 
amplifier,  a  counter,  or  the  like.  Try  to  Identify  these 
groups  and  to  label  what  they  do.  Then  you  can  think 
about  them  together.  Instead  of  worrying  about 

b.  Important  Points 

Are  there  certain  points  In  the  circuit  that  seem  to  be 
particularly  Important?  These  might  be  places  where 
several  of  the  groups  you  just  identified  connect  with 
each  other.  They  might  be  places  where  several  signals 
come  together  or  diverge,  or  where  there  Is  only  a  single 
connection  between  two  parts  of  the  circuit.  Often  they 
lie  on  the  "Information"  paths  that  are  described  above. 
Try  to  decide  what  the  signal  Is  doing  at  these  points. 


HOW  THE  CIRCUIT  RESPONDS 


What  are  the  dynamic  aspects  of  the  circuit?  Try  to  see  what 
happens  when  the  Input  or  the  controls  are  changed.  How  does  this 
affect  the  output?  How  does  It  affect  the  Important  points 
mentioned  above.  When  you  change  the  Input:  Does  the  voltage  go 
up  or  down?  Does  the  current  go  up  or  down? 

Part  2:  Attacking  the  Problem 

MAKE  CHANGES 

Now  turn  to  the  problem  that  you  are  supposed  to  solve.  Start  by 
asking  how  the  changes  you  are  to  make  will  affect  the  parts  of 
the  circuit  that  you  Identified.  What  changes  are  necessary?  If 
the  circuit  does  not  work  and  you  are  to  fix  It,  where  might  the 
problem  be?  What  changes  are  necessary?  How  should  the  final 
circuit  differ  from  what  you  have  to  start  with? 

TRACE  THE  EFFECTS  OF  CHANGES 

When  you  make  a  change  In  the  circuit,  try  to  figure  out  what 
effect  It  will  have  on  the  rest  of  the  circuit. 

5.3  Subjects 

The  sample  of  21  novice  technicians  for  the  third  year  of  work  were 
obtained  from  eight  electronics  trade  schools  In  the  Los  Angeles  area.  A 
novice  was  defined  as  a  student  who  had  completed  one  year  In  an 
electronics  technology  program.  In  addition,  it  was  required  that  each 
subject  could  Interpret  both  analog  and  digital  circuit  block  diagrams. 
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5.4 


Materials 


Four  of  six  circuits  designed  for  the  second  year  of  work  (2  digital,  2 
analog)  and  two  of  the  three  tasks  (fix  error,  alter  circuit)  were  used 
here  again.  This  manipulation  allowed  the  data  from  this  study  to  be 
compared  to  those  generated  by  the  expert  technicians  from  the  second  year 
of  work.  The  specific  circuit  diagrams  used  are  presented  In  Figures  4-1A, 
4-1C,  4-2A,  and  4-2C,  Chapter  4. 


5.5  Design  and  Procedure 

Each  subject  was  assigned  to  one  of  three  conditions  with  seven  subjects 
per  condition.  The  subjects  In  the  Suggested  Guidelines  condition  were 
Instructed  In  the  procedural  steps  presented  In  Section  5.2.  Another  group 
of  subjects  also  was  given  the  guidelines  plus  they  were  forced  to  comply 
with  the  suggested  procedures  (Mandatory  Guidelines  condition).  This  was 
accomplished  through  repeated  reminders  to  attend  to  each  section  of  the 
guidelines.  The  third  group  served  in  an  Activity  Control  condition,  to 
test  the  possibility  that  Improved  performance  with  the  guidelines  could  be 
due  simply  to  Increasing  the  amount  of  time  a  novice  spent  with  the  circuit 
prior  to  problem  solving. 

The  control  group  was  Instructed  to  Increase  Its  activity  relative  to  the 
circuit  diagram  In  the  first  phase.  Subjects  were  given  no  guidance 
regarding  the  second  problem  solving  phase.  The  control  activity  task 
Instructions  were  as  follows: 


Part  1:  Becoming  Familiar  with  the  Circuit 
LIST  THE  PARTS 

Make  a  parts  list  on  a  blank  sheet  of  paper.  Use  a  heading  for 
each  kind  of  component  (resistor,  diode,  etc.)  that  the  circuit 
uses.  Next  to  each  heading  draw  the  symbol  used  to  represent  that 
component.  Below  each  heading  list  the  values  of  that  component 
that  appear  In  the  circuit  and  how  many  of  each  value  are  used 
(for  example,  20  ohm:  3  10K  lhm:  1,  etc.). 

REDRAW  THE  CIRCUIT 


On  a  blank  sheet  of  paper  make  a  copy  of  the  circuit  diagram.  The 
Important  thing  Is  to  copy  the  electronic  logic  of  the  circuit, 
not  necessarily  to  draw  the  exact  picture.  You  may  change  the 
relative  positions  of  the  components  If  you  feel  It  will  make  the 
diagram  clearer.  Use  the  same  labels  for  the  components  as  given 
In  the  original  diagram. 

Part  2:  Attacking  the  Problem 

Now  proceed  to  solve  the  problem. 

Accompanying  both  the  guidelines  and  the  activity  tasks  was  an  example 
circuit  of  comparable  quality  to  the  actual  problems  and  additional  text 
and  diagrams  showing  how  to  carry  out  the  guidelines  of  activity  tasks. 

Each  subject  altered  two  circuits  (one  analog  and  one  digital)  and  fixed 
two  different  circuits  (again,  one  analog  and  one  digital)  under  one  of  the 
three  problem  solving  conditions  (Suggested  Guidelines,  Mandatory 
Guidelines,  or  Activity  Control).  Thus,  four  different  problems 


represented  a  unique  set  In  that  they  were  randomly  selected  without 
replacement  from  a  pool  of  24  different  combinations  of  problem  task, 
circuit  type,  and  testing  order. 

As  In  the  second  year's  experiment,  the  subjects  were  given  considerable 
training  on  the  production  of  verbal  protocols  while  solving  the  problems. 
In  addition  to  explicit  Instructions  prior  to  problem  solving,  they  were 
prompted  during  problem  solving  by  the  experimenter's  use  of  comments  such 
as  "What  are  you  thinking  about  now?"  or  "What  are  you  trying  to  do  now?" 
This  sort  of  prompting  occurred  when  subjects  had  not  spoken  for 
approximately  30  seconds. 

After  telling  the  subject  that  verbal  protocols  were  desired  from  him,  the 
experimenter  read  the  following  Instructions  aloud. 

"First,  let  me  give  you  a  quick  definition  of  a  verbal  protocol. 
Basically,  It  Is  thinking  out  loud.  This  means  that  while  you  are 
solving  the  problems,  you  must  also  be  monitoring  your  own  thoughts 
and  make  them  known  to  us  by  saying  them  aloud. 

Our  purpose  In  having  you  do  this  Is  that  we  are  Interested  both  In 
your  solution  to  each  problem  and  how  you  arrive  at  that  solution. 
This  Is  because  there  are  usually  several  different  ways  to  arrive 
at  any  one  solution.  So  we  will  keep  a  record  of  everything  you  do 
In  coming  up  with  your  solution.  One  way  that  we  will  do  this  Is 
to  have  you  think  out  loud  as  you  are  solving  the  problems.  I 
should  point  out  that  since  most  people  do  not  naturally  think  out 
loud,  this  may  require  some  concentration  on  your  part. 


We  would  like  to  be  able  to  write  down  a  list  of  distinct  steps 
that  you  go  through  In  order  to  arrive  at  your  final  solution.  You 
can  help  us  by  telling  us  what  you  are  thinking  at  each  point  as 
you  work  toward  your  solution,  and  also  why  you  are  thinking  about 
the  steps  you  give  us. 

Do  not  for  any  reason  edit  anything  out  from  your  thinking.  For 
example.  If  you  see  a  step  that  you  would  like  to  carry  out,  but 
you  don't  see  how  to  do  It  right  away,  go  ahead  and  tell  us  what 
you  would  like  to  do  anyway  and  why  you  would  like  to  do  It.  Also, 
If  you  start  to  do  something,  and  then  decide  against  It,  tell  us 
about  It  anyway,  and  why  you  changed  your  mind. 

We  are  not  only  Interested  In  the  steps  that  lead  directly  to  your 
solution,  we  are  also  very  Interested  In  the  things  you  think  about 
and  then  decide  are  wrong.  We  need  a  record  of  everything  you 
think  about  until  you  finally  come  up  with  your  solution." 

Before  presenting  the  problems  to  the  subject,  the  experimenter  gave  him  a 
printed  set  of  guidelines  or  activity  tasks  to  study.  In  addition, 
keywords  from  the  guidelines  or  tasks  were  posted  on  the  wall  directly  in 
front  of  the  subject  to  remind  him  of  the  steps  Involved.  Next,  the 
experimenter  explicitly  demonstrated  how  to  perform  the  presolution  phase. 
Involving  the  guidelines  or  tasks,  using  a  problem  whose  difficulty  level 
was  comparable  to  those  the  subject  would  solve. 

During  the  problem-solution  phase,  the  experimenter  did  not  disrupt  the 
subjects  In  the  Suggested-Guldellnes  or  Activity-Control  conditions  to 
remind  them  to  use  the  guidelines.  In  contrast,  the  experimenter 
repeatedly  reminded  the  subjects  In  the  Forced-Guidelines  condition  to 
attend  to  each  section  of  the  guidelines.  Their  performing  of  each 
guideline  task  was  verified  by  their  verbal  protocols.  An  additional 


measure  taken  with  these  subjects  was  to  withhold  the  statement  of  the 
problem  task  for  a  given  circuit  until  the  presolution  guideline  activities 
were  completed. 

5.6  Results 

The  performance  of  the  technicians  was  compared  between  conditions  In  this 
experiment  (Activity  Control,  Suggested  Guidelines,  Forced  Guidelines)  and 
with  the  performance  of  the  expert  and  novice  technicians  from  the  second 
year  of  this  research  program.  Three  variables  were  studied:  Accuracy  of 
solutions,  classification  of  solution  protocols,  and  solution  time. 

5.6.1  Accuracy  of  Solutions.  Accuracy  data  In  terms  of  the  proportion  of 
problems  for  which  workable  solutions  were  provided  for  each  task  are 
presented  In  Table  5-1.  A  solution  that  was  completely  correct  was 
assigned  one  point;  whereas,  a  solution  that  was  close  to  correct  but 
Incomplete  was  assigned  one-half  point. 

The  pattern  of  results  shown  In  Table  5-1  Is  highly  regular.  The 
performance  of  both  the  Suggested  Guidelines  group  and  the  Forced 
Guidelines  group  was  more  similar  to  the  Experts  from  Year  2  than  to  the 
Novices  (No  Guidelines)  from  Year  2.  Thus,  the  guidelines  appear  to  have 
been  effective:  the  accuracy  of  performance  was  enhanced  by  43%  In  each  of 
the  guidelines  conditions  as  compared  to 

This  conclusion  Is  complicated  somewhat  by  the  outcome  for  the 
Activity-Control  group.  These  subjects  were  not  given  instructions  that 
would  alter  their  problem-structuring  or  problem-solving  behavior,  but 
rather  they  were  asked  to  perform  two  tasks  that  would  Increase  their 
Interaction  with  the  circuits  prior  to  approaching  the  problems.  These 
tasks  were  to  make  a  parts  list  and  to  redraw  the  circuit  In  a  new  layout. 
The  technicians  In  this  condition  exhibited  accuracy  scores  that  also  were 
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FIX 


ALTER 


EXPERT  (YR  2) 

.50 

1.00 

GUIDELINES  (YR  3) 

.52 

.73 

FORCED  GUIDELINES  (YR  3) 

.50 

.75 

NOVICE  (YR  2) 

.20 

.20 

ACTIVITY  CONTROL  (YR  3) 

.63 

.75 

more  similar  to  the  Experts  from  Year  2  than  to  the  Novices  (no  guidelines) 
from  Year  2.  From  this  result,  it  would  appear  that  the  element  of  the 
guidelines  that  promoted  Interaction  with  the  circuits  prior  to  approaching 
the  problem  was  the  key  to  the  enhancement  of  performance. 

An  alternative  hypothesis  that  might  explain  this  result  is  that  the  skill 
level  of  the  subject  population  was  significantly  different  in  the  Year  3 
sample  from  that  of  the  Novice  group  from  the  Year  2  possibility.  This 
sample  does  not  seem  likely  given  that  the  subjects  were  obtained  from  the 
same  electronics  trade  schools  and  they  were  at  the  same  stage  in  their 
education. 

5.6.2  Classification  of  Protocols.  As  In  the  second  year  of  work,  the 
transcribed  protocols  first  were  Inspected  for  patterns  by  one  of  the 
project's  Principal  Investigators.  This  time,  however,  the  protocols  also 
were  inspected  for  patterns  by  a  research  assistant  such  that  a  measure  of 
inter-judge  reliability  could  be  computed.  Each  judge  worked  Independently 
and  was  blind  to  the  condition  assignment  of  each  subject.  All  of  the 
protocols  were  described  using  the  four  classifications  developed  in  the 
Year  2  work:  namely  "Directed,"  "re-directed,"  "immediate,"  and 

"trlal-and-error."  Each  protocol  was  classified  systematically  based  on 
clear  breaks  In  the  solution  paths.  If-then  statements,  and  admissions  of 
Incorrect  attempts  and  guessing.  The  two  judged  agreed  on  the 
classification  of  84%  of  the  protocols;  the  remaining  Instances  were 
discussed  by  the  judges  to  achieve  100%  agreement. 

The  classification  results  are  presented  in  Table  5-2.  As  was  the  case 
with  the  accuracy  data,  all  three  groups  from  the  third  years  work 
performed  more  like  the  Experts  than  the  Novices  (no  guidelines)  from  Year 
2.  These  protocols  were  predominantly  classified  as  (56%)  as  "directed" 
behavior  —  the  highest  level  of  cognitive  performance.  And,  much  of  the 
balance  (31%)  were  classified  as  "re-directed"  behavior.  In  contrast,  the 


TABLE  5-2 


Proportion  Of  Protocols  Falling  Into 
Each  Classification  As  A  Function  Of 
Skill  Level  And  Condition 


DIRECTED 
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IMMEDIATE 

TRIAL  AND 

ERROR 

EXPERT  (YR  2) 

.56 

.28 

.05 

.11 

GUIDELINES  (YR  3) 

.58 

.33 

.00 

.08 

FORCED  GUIDELINES  (YR  3) 

.50 

.25 

.15 

.10 

NOVICE  (YR  2) 

.33 

.23 

.13 

.31 

ACTIVITY  CONTROL  (YR  3) 

.57 

.31 

.06 

.06 

Novices  with  no  guidelines  from  Year  2  showed  substantial  "trial -and-error" 
behavior.  Thus,  while  the  guidelines  derived  from  the  problem-solving 
protocols  of  the  Experts  were  successful  for  altering  the  behavior  patterns 
of  novices,  so  were  the  activity  tasks  given  to  the  control  subjects.  This 
again  suggests  that  the  effective  component  of  the  procedural  guidelines 
was  the  enhanced  Interaction  with  the  circuit  prior  to  addressing  the 
problem.  This  activity  appears  to  have  been  Instrumental  In  disrupting  the 
unsystematic  matching  of  the  current  problem  with  textbook  examples  that 
would  characterize  the  novices'  behavior  as  "trial -and-error"  or 
"Immediate." 

5.6.3  Solution  Time.  Table  5-3  presents  the  average  problem  solution 
time  as  a  function  of  problem  and  skill  level  or  condition.  Most  striking 
Is  the  differences  overall  between  Year  2  and  Year  3.  Although  the  two 
guideline  groups  and  the  Activity-control  group  from  Year  3  performed  like 
the  Experts,  both  In  terms  of  accuracy  and  behavior  pattern,  these  subjects 
required  considerably  longer  times  to  arrive  at  their  solutions.  This 
suggests  that  the  guidelines  and  activity  tasks  altered  the 
problem-solution  behaviors  In  ways  that  were  effective,  but  were  not  used 
routinely  by  the  Novice  technicians. 

5.7  Conclusions 

Without  guidance  (Year  2),  Novice  technicians  exhibited  In  systematic, 
"trlal-and-error"  or  "Immediate"  solution  behavior  toward  solving  problems 
with  circuits;  whereas.  Expert  technicians  showed  directed  behavior  that  Is 
characterized  by  an  apparent  plan  toward  a  solution,  with  goals  and  If-then 
operations.  This  difference  would  be  relatively  uninteresting  If  It  were 
simply  an  ephl phenomenon  of  skill  level  rather  than  a  causal  Influence. 
However,  with  guidance  to  first  understand  the  circuit  before  Initiating  a 
solution  sequence,  and  then  to  map  out  a  goal-directed  strategy,  the 
behavior  patterns  of  Novices  resembled  those  of  Experts  with  an 
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accompanying  Increase  In  the  quality  of  the  solutions.  Thus,  the  results 
from  the  third  year  of  work  suggest  that  at  least  some  major  differences  In 
the  quality  of  performance  by  Expert  versus  Novice  technicians  lie  In  their 
patterns  of  problem-solving  behavior. 

Furthermore,  the  differences  In  problem-solving  behavior  appear  to 
represent  a  production  deficiency.  That  Is,  technicians  who  simply  were 
forced  to  perform  activities  with  the  circuits  before  addressing  the 
problems  also  performed  more  like  Experts  than  uninstructed  Novices,  both 
In  terms  of  accuracy  and  the  pattern  of  problem-solving  behavior.  Novices 
not  specifically  told  to  first  understand  the  circuit  or  to  establish  a 
goal-directed  plan  did  so  In  any  case  as  a  result  of  having  constructed  a 
parts  list  and  re-drawing  the  circuit  layout.  Thus,  the  effective 
behaviors  already  were  a  part  of  the  Novices'  repertoires,  but  In  the 
absence  of  required  preliminary  activities  with  the  circuits,  a  form  of 
template  matching  of  the  current  problem  with  previous  examples  In  memory 
was  employed. 

It  seems  reasonable  that  the  activity  tasks  enhanced  the  subjects' 
understanding  of  the  circuits,  and  thereby  Induced  a  systematic  plan  for 
solution.  The  solution-time  data  suggest  that  the  Novices  use  of 

systematic  plans  Is  not  routine,  as  they  required  more  time  to  arrive  at 
the  solutions  than  the  Experts. 


6.  GENERAL  CONCLUSIONS 


The  purpose  of  this  program  of  research  was  to  Identify  differences  in 
knowledge  that  are  characteristic  of  the  distinction  between  Expert  and 
Novice  technicians,  and  to  validate  the  explanatory  power  of  these 
differences  In  controlled  experiments.  The  first  experiment  (Year  1)  was 
designed  to  examine  differences  between  skill  levels  In  the 
structural /functional  knowledge  base.  That  experiment  revealed  great 
Individual  differences  among  technicians  In  structural /functional  knowledge 
as  Inferred  from  performance  of  circuit  reconstruction  and  circuit 
partitioning  tasks.  These  Individual  differences  overshadowed  most 
differences  between  Experts  and  Novices.  Nevertheless,  the  Expert-Novice 
distinction  was  confirmed  In  both  accuracy  and  performance  time  data.  It 
was  therefore  concluded  that  the  Interesting  differences  between  Experts 
and  Novices  do  not  lie  In  structural /functional  knowledge  of  circuits. 

The  second  experiment  (Year  2)  revealed  clear  differences  between  Experts 
and  Novices  on  procedural  knowledge  on  how  to  fix,  alter,  complete 
circuits.  These  differences,  which  were  quantified  through  analyses  of 
recorded  problem  solution  protocols,  were  translated  Into  guidelines  for 
troubleshooting  and  altering  circuits.  In  particular.  Experts  reliably 
tried  to  understand  the  circuit  before  Initiating  a  solution  sequence  and 
the  solution  sequence  was  systematic  and  apparently  goal -directed. 

In  the  third  experiment  (Year  3),  the  problem-solving  behavior  of  Novices 
was  successfully  changed  such  that  they  mimicked  the  Experts  from 
Experiment  2  on  their  procedural  patterns  for  troubleshooting  and  altering 
circuits.  Correspondingly,  the  quality  of  the  solutions  was  Improved 
dramatically,  by  43%.  Thus,  the  systematic  problem  solving  behavior  of  the 
Experts  appears  not  be  have  been  an  eplphenomenon  of  their  expertise,  but 
rather  It  seems  to  be  an  Integral  component  of  their  expertise. 


In  addition.  It  appears  that  the  Novices  were  capable  of  generating  the 
effective  behaviors  on  their  own  when  they  were  required  to  Interact  with  a 
circuit  prior  to  Initiating  a  problem-solution  sequence.  Thus,  one 
Important  stage  In  the  transmission  from  the  Novice  to  the  Expert  level  of 
expertise  Is  the  disruption  of  a  strategy  where  the  current  problem  Is 
matched  unsystematically  with  previous  examples  in  memory,  such  that  a  more 
systematic  approach  Is  developed  and  executed  based  on  an  understanding  of 
the  circuit.  This  component  of  the  transition  from  Novice  to  Expert  levels 
of  skill  has  been  Identified  In  other  domains  of  knowledge.  Including 
computer  programming  (Adelson,  1984)  and  chess  (Chase  &  Simon,  1973),  where 
Experts  are  said  to  form  abstract  representations  of  problem  spares  while 
Novices  rely  on  more  concrete  representations. 

The  present  results  suggest  that  the  problem-solving  behavior  and 
performance  of  Novice  technicians  can  be  Improved  considerably  by  requiring 
interaction  with  the  elements  and  layout  of  a  circuit  diagram  prior  to  the 
Initiation  of  a  solution  sequence.  This  finding  should  help  in  providing 
guidelines  for  training  technicians  to  better  understand  and  troubleshoot 
equipment. 
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